Semi-supervised Classification Using Local and Global Regularization

In this paper, we propose a semi-supervised learning (SSL) algorithm based on local and global regularization. In the local regularization part, our algorithm constructs a regularized classifier for each data point using its neighborhood, while the global regularization part adopts a Laplacian regularizer to smooth the data labels predicted by those local classifiers. We show that some existing SSL algorithms can be derived from our framework. Finally we present some experimental results to show the effectiveness of our method. Introduction Semi-supervised learning (SSL), which aims at learning from partially labeled data sets, has received considerable interests from the machine learning and data mining communities in recent years (Chapelle et al., 2006b). One reason for the popularity of SSL is because in many real world applications, the acquisition of sufficient labeled data is quite expensive and time consuming, but the large amount of unlabeled data are far easier to obtain. Many SSL methods have been proposed in the recent decades (Chapelle et al., 2006b), among which the graph based approaches, such as Gaussian Random Fields (Zhu et al., 2003), Learning with Local and Global Regularization (Zhou et al., 2004) and Tikhonov Regularization (Belkin et al., 2004), have been becoming one of the hottest research area in SSL field. The common denominator of those algorithms is to model the whole data set as an undirected weighted graph, whose vertices correspond to the data set, and edges reflect the relationships between pairwise data points. In SSL setting, some of the vertices on the graph are labeled, while the remained are unlabeled, and the goal of graph based SSL is to predict the labels of those unlabeled data points (and even the new testing data which are not in the graph) such that the predicted labels are sufficiently smooth with respect to the data graph. One common strategy for realizing graph based SSL is to minimize a criterion which is composed of two parts, the first part is a loss measures the difference between the predictions and the initial data labels, and the second part is a smoothness penalty measuring the smoothness of the predicted labels over the whole data graph. Most of the past Copyright c © 2008, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. works concentrate on the derivation of different forms of smoothness regularizers, such as the ones using combinatorial graph Laplacian (Zhu et al., 2003)(Belkin et al., 2006), normalized graph Laplacian (Zhou et al., 2004), exponential/iterative graph Laplacian (Belkin et al., 2004), local linear regularization (Wang & Zhang, 2006) and local learning regularization (Wu & Schölkopf, 2007), but rarely touch the problem of how to derive a more efficient loss function. In this paper, we argue that rather than applying a global loss function which is based on the construction of a global predictor using the whole data set, it would be more desirable to measure such loss locally by building some local predictors for different regions of the input data space. Since according to (Vapnik, 1995), usually it might be difficult to find a predictor which holds a good predictability in the entire input data space, but it is much easier to find a good predictor which is restricted to a local region of the input space. Such divide and conquer scheme has been shown to be much more effective in some real world applications (Bottou & Vapnik, 1992). One problem of this local strategy is that the number of data points in each region is usually too small to train a good predictor, therefore we propose to also apply a global smoother to make the predicted data labels more comply with the intrinsic data distributions. A Brief Review of Manifold Regularization Before we go into the details of our algorithm, let’s first review the basic idea of manifold regularization (Belkin et al., 2006) in this section, since it is closely related to this paper. As we know, in semi-supervised learning, we are given a set of data points X = {x1, · · · ,xl,xl+1, · · · ,xn}, where Xl = {xi}i=1 are labeled, and Xu = {xj}j=l+1 are unlabeled. Each xi ∈ X is drawn from a fixed but usually unknown distribution p(x). Belkin et al. (Belkin et al., 2006) proposed a general geometric framework for semisupervised learning called manifold regularization, which seeks an optimal classification function f by minimizing the following objective Jg = ∑l i=1 L(yi, f(xi,w)) + γA‖f‖F + γI‖f‖I, (1) where yi represents the label of xi, f(x,w) denotes the classification function f with its parameter w, ‖f‖F penalizes the complexity of f in the functional space F , ‖f‖I reflects Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008)